System and method for effective section detecting of hand gesture

ABSTRACT

A system is provided for detecting an effective section of a gesture by recognizing the gesture, pose information and motion information included in the gesture from an acquired image. In addition, a controller determines whether a pose has been recognized based on the pose information and when the pose has been recognized, an effective section is detected based on a start point and an end point of the pose. Further, when the effective section for the pose is detected, the gesture is recognized based on the motion information.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Korean PatentApplication No. 10-2012-0148597 filed in the Korean IntellectualProperty Office on Dec. 18, 2012, the entire contents of which areincorporated herein by reference.

BACKGROUND

(a) Field of the Invention

The present invention relates to a system and a method for detecting aneffective section of a hand gesture.

(b) Description of the Related Art

A user's intention of a gesture is important in a system that recognizesa gesture, based on images collected by an imaging device (e.g., acamera) due to difficulty in accurately determining gestures when thesystem recognizes actions other than predetermined gestures.

Further, for a system using a method of recognizing gestures by signalprocessing in common systems for detecting and recognizing gestures, thesystem recognize a gesture from when there is displacement of apredetermined motion, thus, decreasing the speed at which the gesture isrecognized. In addition, the rate of recognizing a gesture decreases,when the size of a motion trajectory is different from the size of apredetermined trajectory. A gesture is recognized using a learningalgorithm and the trajectory of the gesture may not fit to apredetermined trajectory, to classify an incorrect recognition.Accordingly, a user may have to stop the motion at the beginning and theend of a gesture to allow the system to recognize the type of gesturebeing performed by the user. The above information disclosed in thissection is only for enhancement of understanding of the background ofthe invention and therefore it may contain information that does notform the prior art that is already known in this country to a person ofordinary skill in the art.

SUMMARY

The present invention provides a system and a method for detecting aneffective section of a hand gesture having advantages of improving theaccuracy of detecting an intention of a hand gesture. An exemplaryembodiment of the present invention provides a method for detecting aneffective section (e.g., a section from a start point to an end point ofa recognizable gesture) of a hand gesture using a system that detects aneffective section of a hand gesture to recognize a gesture.

The method may include acquiring an image that includes a gesture and abackground; extracting pose information and motion information includedin the gesture, as features, from the gesture in the image; determiningwhether a pose has been recognized based on the pose information;detecting an effective section of the pose based on a start point (e.g.,the motion of the pose when the gesture is first performed) and an endpoint of the pose (e.g., the motion of the pose when the gesture iscompleted), when the pose has been recognized; and recognizing thegesture based on the motion information, when an effective section forthe pose has been detected.

The determining of whether the pose has been recognized may include:determining which of one or more items of feature information, stored inadvance, the extracted pose information corresponds to; and recognizingthe extracted pose information as a pose, when the extracted poseinformation corresponds to any one of the feature information.

The detecting of an effective section may include: determining the startpoint of the pose; determining the end point of the pose; determiningwhether the section from the start point to the end point continues fora predetermined time or more; and determining the section between thestart point to the end point as an effective section, when the sectioncontinues for the predetermined time or more.

The recognizing of a gesture may include: determining which of one ormore items of feature information, stored in advance, the extractedmotion information corresponds to; and outputting the extracted motioninformation as a gesture recognition result, when the extracted motioninformation corresponds to any one of the feature information.

The extracting of pose information and motion information as featuresmay include extracting the pose as pose information using a templatematching algorithm or extracting and then modeling the features of apose image, and the motion as motion information by recognizing themotion, using any one of signal processing or learning algorithm.

Another exemplary embodiment of the present invention provides a systemthat detects an effective section of a hand gesture.

The system includes and plurality of units executed by a controller thathas a processor and a storage unit. The plurality of units may includean image acquiring unit that acquires an image including a backgroundand a gesture having pose information and motion information; a featuredetecting unit that detects the pose information and the motioninformation from the gesture and extracts the pose information bydetermining whether there is information that corresponds to the motioninformation in previously feature information stored; a gesture startpoint determining unit that detects the start point of the poseinformation, when the feature detecting unit detects the poseinformation; a gesture end point determining unit that detects the endpoint of the pose information, when the feature detecting unit detectsthe pose information; and a gesture recognizing unit that detects aneffective section for the pose based on the start point detected by thegesture start point determining unit and the end point detected by thegesture end point determining unit, and recognizes a gesture bydetermining the motion information extracted by the feature detectingunit is included in the previously feature information stored, when theeffective section is detected.

The plurality of units may further include: a background removing unitthat removes the background from the image acquired by the imageacquiring unit and provides only the gesture to the feature detectingunit; and a feature information storing unit that stores and providesfeature information used by the feature detecting unit and the gesturerecognizing unit to determine the pose information and the motioninformation.

According to the present invention, since it may be possible to improveaccuracy in detecting the start and the end of a gesture, it may bepossible to remove an incorrect recognition of a non-intended gesture bythe system. Further, since a natural gesture may be set, the gesturerecognition system may be intuitively used.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary diagram for recognizing a common hand gestureaccording to the related art;

FIG. 2 is an exemplary diagram illustrating the structure of a systemthat detects an effective section of a gesture according to an exemplaryembodiment of the present invention;

FIG. 3 is an exemplary flowchart illustrating a method of detecting aneffective section according to an exemplary embodiment of the presentinvention; and

FIG. 4 is an exemplary diagram illustrating a combination of poserecognition and gesture motion recognition according to an exemplaryembodiment of the present invention.

DETAILED DESCRIPTION

It is understood that the term “vehicle” or “vehicular” or other similarterm as used herein is inclusive of motor vehicles in general such aspassenger automobiles including sports utility vehicles (SUV), buses,trucks, various commercial vehicles, watercraft including a variety ofboats and ships, aircraft, and the like, and includes hybrid vehicles,electric vehicles, combustion, plug-in hybrid electric vehicles,hydrogen-powered vehicles and other alternative fuel vehicles (e.g.fuels derived from resources other than petroleum).

Although exemplary embodiment is described as using a plurality of unitsto perform the exemplary process, it is understood that the exemplaryprocesses may also be performed by one or plurality of modules.Additionally, it is understood that the term controller refers to ahardware device that includes a memory and a processor. The memory isconfigured to store the modules and the processor is specificallyconfigured to execute said modules to perform one or more processeswhich are described further below.

Furthermore, control logic of the present invention may be embodied asnon-transitory computer readable media on a computer readable mediumcontaining executable program instructions executed by a processor,controller or the like. Examples of the computer readable mediumsinclude, but are not limited to, ROM, RAM, compact disc (CD)-ROMs,magnetic tapes, floppy disks, flash drives, smart cards and optical datastorage devices. The computer readable recording medium can also bedistributed in network coupled computer systems so that the computerreadable media is stored and executed in a distributed fashion, e.g., bya telematics server or a Controller Area Network (CAN).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof, As used herein, the term “and/or”includes any and all combinations of one or more of the associatedlisted items.

In the following detailed description, only exemplary embodiments of thepresent invention have been shown and described, simply by way ofillustration. As those skilled in the art would realize, the describedembodiments may be modified in various different ways, all withoutdeparting from the spirit or scope of the present invention.Accordingly, the drawings and description are to be regarded asillustrative in nature and not restrictive. Like reference numeralsdesignate like elements throughout the specification. Throughout thespecification, unless explicitly described to the contrary, the word“comprise” and variations such as “comprises” or “comprising,” will beunderstood to imply the inclusion of stated elements but not theexclusion of any other elements.

Hereinafter, a system and a method for detecting an effective section(e.g., a section from a start point to an end point of a recognizablegesture) of a hand gesture according to an exemplary embodiment of thepresent invention will be described with reference to the accompanyingdrawings. FIG. 1 is an exemplary diagram for recognizing a common handgesture.

As shown in (a) of FIG. 1, according to a common technology ofrecognizing a gesture, a system recognizes a motion trajectory of a handfeature by signal processing or using a learning algorithm. However,common systems for detecting and recognizing a gesture are n providedwith a module that detects the accurate point of e where a user inputs agesture.

Specifically, when a gesture is recognized by signal processing, thegesture may begin to be recognized from when there is displacement in apredetermined motion, thus decreasing the speed at which a gesture maybe recognized. Further, the rate of recognizing a gesture decreases,when the size of a motion trajectory is different from the size of apredetermined trajectory.

When recognition is performed using a learning algorithm and apredetermined trajectory shown in (b) of FIG. 1 does not fit to thetrajectory of an actual gesture of a user shown in (c) of FIG. 1, thesystem may be configured to determine an incorrect recognition of theuser's gesture. Accordingly, a user has to stop the motion at the startand the end of the gesture. In other words, since the start and the endof a gesture motion are not accurately recognized, uncertainty of thesystem recognizing the motion increases. Further, when the userunintentionally makes an action similar to a predetermined gesture, thesystem may recognize the action as a gesture, causing frequent incorrectrecognition. Therefore, according to an exemplary embodiment of thepresent invention, it may be possible to reduce incorrect recognition ofa gesture, due to increased accuracy in detecting the start and the endof a gesture

An exemplary embodiment of the present invention uses a system thatdetects an effective section of a gesture, as shown in FIG. 2, toaccurately determine the start and the end of a gesture. In an exemplaryembodiment of the present invention, an effective section may bedetermined by combining pose recognition with motion recognition toaccurately determine the start and the end of a gesture. FIG. 2 is anexemplary diagram illustrating the structure of a system that detects aneffective section according to an exemplary embodiment of the presentinvention.

As shown in FIG. 2, a system 100 that detects an effective section mayinclude a plurality of units executed by a controller. The plurality ofunits may include an image acquiring unit 110, a background removingunit 120, a feature detecting unit 130, a feature information storingunit 140, a gesture start point determining unit 150, a gesture endpoint determining unit 160, and a gesture recognizing unit 170.

The image acquiring unit 110 may be configured to acquire images inputvia a device capable of acquiring images, such as a camera (not shown).The images may include not only a user's gesture, but also thebackground image of the user's gesture. The background removing unit 120may be configured to acquire the user's gesture by removing thebackground from the image acquired by the image acquiring unit 110.There are many methods of removing the background from an image and anexemplary embodiment of the present invention does not specify any onemethod.

The feature detecting unit 130 may be configured to detect a feature forthe user's gesture from the user's gesture acquired by the backgroundremoving unit 120 and feature information stored in the featureinformation storing unit 140. The feature detecting unit 130 may beconfigured to recognize the pose of a hand from the user's gesture.Specifically, the pose may be recognized by comparing the entire imagesuch as a template matching algorithm or by extracting or modeling afeature on a pose image, such that any one of them is not specified inthe description.

The feature detecting unit 130 may be configured to recognize the motionof the hand, in response to recognizing the hand pose. The motion of thehand may be recognized by signal processing or a learning algorithm andthe method of recognizing the motion of the hand is well known in theart; therefore, it is not described in detail herein. However, whenfailing to recognize the hand pose, the feature detecting unit 130 maybe configured to detect the feature until the hand pose is recognized.The feature may be the pose and motion of the user's hand, in which thepose may he the shape of the hand and the motion may be the movementdirection or movement type of the hand.

The feature information storing unit 140 may be configured to store aplurality of predetermined items of feature information and provide thefeature information to the feature detecting unit 130 and the gesturerecognizing unit 170. The feature information may include poseinformation and motion information. The pose information may includeinformation regarding the poses such as a fist with only the thump orthe forefinger stretched out, or the palm open upside, or the fingerspursed. The motion information may include information on variousmotions such as moving a hand up and down or left and right, rotating ahand, and waving a hand.

When the feature detecting unit 130 detects features of the user's hand,the gesture start point determining unit 150 may be configured todetermine the start point of the gesture from the motion of the detectedfeatures. When the feature detecting unit 130 detects features of theuser's hand, the gesture end point determining unit 160 may beconfigured to determine the end point of the gesture from the motion ofthe detected features.

The gesture recognizing unit 170 may be configured to receive theinformation on the start point of the gesture determined by the gesturestart point determining unit 150 and the end point of the gesturedetermined by the gesture end point determining unit 160 and determinewhether it is possible to determine the section from the start point tothe end point of the gesture as an effective section of the gesture. Thegesture recognizing unit 170 may be configured to determine theeffective section of the gesture, when the gesture continues for apredetermined time.

When the effective section is determined, the user's gesture may berecognized from the pose of the hand determined by the feature detectingunit 130. Further, the e controller may be configured to determinewhether a motion substantially the same as the motion detected by thegesture recognizing unit 170 is in the motion information stored in thefeature information storing unit 140. When the same information isstored in the feature information storing unit 140, the gesturerecognizing unit 170 may be configured to output the gesture recognitionresult to allow a predetermined control to be performed on therecognized gesture. However, when there the substantially same motioninformation is not detected, the feature detecting unit 130 may beoperated by the controller to detect a feature from the gesture.

A method of detecting an effective section of a gesture using the system100 that detects an effective section is described hereafter withreference to FIG. 3. FIG. 3 is an exemplary flowchart illustrating amethod of detecting an effective section according to an exemplaryembodiment of the present invention.

As shown in FIG. 3, when the image acquiring unit 110 acquires an imagethat includes a user's gesture and the background of the gesture (S100),the background removing unit 120 may be configured to remove thebackground from the image (S110). The method of operating the backgroundremoving unit 120 by the controller to remove the background from animage with a gesture left is known in the art, such that the detaileddescription is not provided herein. When the background removing unit120 removes the background, the feature detecting unit 130 may beconfigured to detect features from the gesture. The features may includethe pose and motion of a hand of the user making the gesture.

First, the feature detecting unit 130 may be configured to determinewhether the information that corresponds to the information regardingthe pose of the detected features corresponds to any one item ofinformation from a plurality of items of information stored in thefeature information storing unit 140, that is, whether the pose has beenrecognized (S130). When the pose of the user's hand does not correspondto any one of the items of feature information stored in the featureinformation storing unit 140, the feature detecting unit 130 may beconfigured to detect features from the next image.

In contrast, when the pose of the user's hand corresponds to any one ofthe items of feature information stored in the feature informationstoring unit 140, the gesture start point determining unit 150 may beconfigured to detect the start point of the motion of the hand and thegesture end point determining unit 160 may be configured to detect theend point of the motion, and then the information on the start point andend point may be transmitted to the gesture recognizing unit 170. Thegesture recognizing unit 170 may be configured to determine whether itis possible to determine the corresponding section as an effectivesection for gesture recognition, based on the e information on the startpoint and end point of the gesture (S140).

When the gesture recognizing unit 170 does not determine the section asan effective section due to discontinuing of the section before apredetermined time, the detecting of features in S120 may be repeated.However, when the pose continues over the predetermined time and thegesture recognizing unit 170 may be configured to determine the sectionas an effective section and the motion of the hand of the gesturedetected by the feature detecting unit 130 may be recognized (S150). Thefeature detecting unit 130 may recognize the motion of a hand usingvarious methods, such as by signal processing or using a learningalgorithm, such that any one of the methods is not specified in thedescription of an exemplary embodiment of the present invention.

The gesture recognizing unit 170 may be configured to determine whetherone or more of the hand motions recognized in S150 is included in thefeature information storing unit 140 is recognized as a gesture (S160).When the same information as the information on the motion of the useris not stored in the feature information storing unit 140, therecognizing of a pose in S130 may be repeated. When the same informationas the information on the motion of the user is stored in the featureinformation storing unit 140, the gesture recognizing unit 170 may beconfigured to output the gesture recognition result and request devicecontrol according to the gesture recognition result (S170).

An exemplary embodiment of the method described above is described withreference to FIG. 4. As an example, an audio system of a vehicle may beoperated based on a pose and a gesture. FIG. 4 is an exemplary diagramillustrating a combination of pose recognition and gesture motionrecognition according to an exemplary embodiment of the presentinvention.

In an exemplary embodiment of the present invention five exemplary itemsof information are stored as pose information and three exemplary itemsof information are stored as motion information in the featureinformation storing unit 140, as shown in FIG. 4. In addition, the poseinformation, an audition system of a vehicle may be activated, when twofingers are stretched out in a V-shape, and the channel button of theaudio system may be operated when a wave motion is performed with thepalm open.

Therefore, as shown in FIG. 4, when a user performs a wave motion withfingers stretched out in a V-shape, the image acquiring unit 110 may beconfigured to acquire an image that includes the gesture and thebackground. Then, the background removing unit 120 may be configured toremove the background from the acquired image. Since it is described inthe exemplary embodiment of the present invention that an audio systemof a vehicle may be activated, as an example, he background in the imagemay be the seats within the vehicle or other environments within thevehicle.

When the background removing unit 120 acquires only the gesture afterremoving the background, the feature detecting unit 130 may beconfigured to detect the pose and the motion of the user's hand whichmay be the features of the gesture. The feature detecting unit 130 maybe configured to determine whether information on the same pose as thedetected pose is stored in the feature information storing unit 140, asshown in FIG. 4.

When feature information on the same pose is stored in the featureinformation storing unit 140, the gesture start point determining unit150 may be configured to detect the start point of the pose of the handand the gesture end point determining unit 160 may be configured todetect the end point of the pose of the hand. After the gesture startpoint determining unit 150 and the gesture end point determining unit160 detect the start time point and the end time point of the pose ofthe hand, the gesture recognizing unit 170 may be configured todetermine whether it is possible to determine the corresponding sectionas an effective section of the gesture. The effective section may bedetermined as an effective section, when the substantially same handpose continues for a predetermined time.

When the effective section is determined, the gesture recognizing unit170 may be configured to recognize the motion of the user. The motionmay be recognized using various methods such as by signal processing ora learning algorithm. When the gesture recognizing unit 170 determinesthat the motion of the user is a wave, the gesture recognition resultmay be transmitted to a controller of the audio system to operate thechannel button of the audio system in a predetermined way and the audiosystem may be operated.

As described above, it may be possible in an exemplary embodiment of thepresent invention to reduce incorrect recognition of a gesture byfinding an effective section of the gesture in gesture recognition.

While this invention has been described in connection with what ispresently considered to be exemplary embodiments, it is to be understoodthat the invention is not limited to the disclosed embodiments, but, onthe contrary, is intended to cover various modifications and equivalentarrangements included within the spirit and scope of the accompanyingclaims.

What is claimed is:
 1. A method for detecting an effective section of agesture, the method comprising: acquiring, by a controller, an imagethat includes the gesture and a background; extracting, by thecontroller, pose information and motion information included in thegesture as features; determining, by the controller, whether a pose hasbeen recognized based on the pose information; in response todetermining that a pose has been recognized, detecting, by thecontroller, the effective section of the gesture based on a start pointand an end point of the pose; and in response to detecting the effectivesection of the gesture, recognizing, by the controller, the gesturebased on the motion information.
 2. The method of claim 1, wherein thepose information includes information regarding the pose of a fist witha thumb or a forefinger stretched out, a palm open upside, or clenchedfingers.
 3. The method of claim 1, ,wherein the motion informationincludes information regarding various motions including moving a handhorizontally or vertically, rotating the hand, and waving the hand. 4.The method of claim 1, wherein the determining of whether the pose hasbeen recognized includes: determining, by the controller, stored featureinformation that corresponds to the extracted pose information; and inresponse to determining corresponding stored feature information,recognizing, by the controller, the extracted pose information as apose.
 5. The method of claim 1, wherein the detecting of the effectivesection of the gesture includes: detecting, by the controller, the startpoint of the pose; detecting, by the controller, the end point of thepose; determining, by the controller, whether the section from the startpoint to the end point continues for a predetermined time or more; andin response to determining that the section continues for thepredetermined time or more, determining, by the controller, the sectionbetween the start point to the end point as the effective section. 6.The method of claim 1, wherein the recognizing of the gesture includes:determining, by the controller, stored feature information thatcorresponds to the extracted motion information; and in response todetermining corresponding stored feature information, outputting, by thecontroller, the extracted motion information as a gesture recognitionresult.
 7. The method of claim 1, wherein the extracting of poseinformation and motion information as features includes: extracting, bythe controller, the pose as pose information using a template matchingalgorithm or extracting and then modeling the features of a pose image,and extracting, by the controller, the motion as motion information byrecognizing the motion using signal processing or learning algorithm, 8.The method of claim 1, wherein the acquiring of the image includes:removing, by the controller, the background from the acquired image. 9.A system that detects an effective section of a hand gesture, the systemcomprising: a controller configured to: acquire an image having abackground and the gesture that includes pose information and motioninformation; detect the pose information and the motion information fromthe gesture; extract the pose information by determining whetherinformation corresponding to the motion information is in stored featureinformation; detect a start point of the pose information, in responseto detecting the pose information; detect an end point of the poseinformation, in response to detecting the pose information; and detectthe effective section for the pose based on the start point and the endpoint; and recognize a gesture by determining whether the extractedmotion information is included in the stored feature information, inresponse to detecting the effective section.
 10. The system of claim 9,wherein the controller is further configured to: remove the backgroundfrom the acquired image; and store and provide feature information usedto detect the pose information and the motion information.
 11. Thesystem of claim 9, wherein the pose information includes informationregarding the pose of a fist with a thumb or a forefinger stretched out,a palm open upside, or clenched fingers.
 12. The system of claim 9,wherein the motion information includes information regarding variousmotions including moving a hand horizontally or vertically, rotating thehand, and waving the hand.
 13. A non-transitory computer readable mediumcontaining program instructions executed by a processor or controller,the computer readable medium comprising: program instructions thatacquire an image having a background and the gesture that includes poseinformation and motion information; program instructions that detect thepose information and the motion information from the gesture; programinstructions that extract the pose information by determining whetherinformation corresponding to the motion information is in stored featureinformation; program instructions that detect a start point of the poseinformation, in response to detecting the pose information; programinstructions that detect an end point of the pose information, inresponse to detecting the pose information; and program instructionsthat detect the effective section for the pose based on the start pointand the end point; and program instructions that recognize a gesture bydetermining whether the extracted motion information is included in thestored feature information, in response to detecting the effectivesection.
 14. The non-transitory computer readable medium of claim 13,further comprising: program instructions that remove the background fromthe acquired image; and program instructions that store and providefeature information used to detect the pose information and the motioninformation.